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1 n OBJECTIVES 

The staled objectives for the Summer of 1993 were; 

1. Review Ite liRlividual Development Plan Snmys for 1994 in order to automate the analysis of 
the Needs Assessment elTort. 

2. Develop and impleme.it evaluation methodologies to perform ongoing program-wide course-to- 
course assessmcRl. This includes. 

. Propose a methodology to develop and implement objective, performance-based, 

a<;<;pssmenl instruments for each training efTort. . 

. Mechanize course evaluation forms and develop software to facilitate the data gathering, 

analysis and reporting processes. , 

. ?mple.nent the methodology, forms, and software in at least one training course or 

seminar selected among those normally offered in the summer at KSC. 

Section 2 of this report addresses the work done in regard to the Individual 
Surveys for 1994. Section 3 presents the methodology proposed to ' 

performance-based, assessment instruments for each training course offered at KSC. 
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7 0 THE INDIVIDUAL DEVELOPMENT PLAN S URVEYS FOR 1994 


Section 2 of this report deals with the first objective, the Individual Develop^t Plan Surveys 
for 1994 The inTormation stored in the Personnel Training Developmental Syrtem (PTOS) was 
[Is^red to a Windows platform and processed using FoxPro for Windows. Section 2.1 
S^ co«ds used to query the data bases and produce the requested r^rts A^n example, the first 
Lge o“report is prLnTed in the ne.xt pages. The full reports, already delivered to NASA a^ 
wSsidered part^f this Final Report. Section 2.2 discusses issues associated to training cost and priority 
and presents recommendations for further research. 


2.1 Commands Used to Query the Data Bases 

2.1.1 Report #1 -- Courses by Type of Training by Organization 


SELECT DISTINCT Report 1.unit, Report 1.tot, Reportl.course_num„ 
COUNT{Reportl.course_num), SUM(Reportl.cost), Courses.title,, 
Traintyp.type, Units.unitname; 

FROM Report 1, Courses, Units, Traintyp; 

WHERE Courses.coursenum = Report l.course_num; 

AND Units.unit = Rcporll.unit; 

AND Trainlyp.tot = Rcporll.tol; 

AND Reportl.unit = "AC"; 

GROUP BY Rcportl .courscjium; 

ORDER BY Reportl tot; 


S?SRW>rtl.rrx FROM Rtportl WIDTH 0 COLUMN NOOVERWROB 
REPORT FORM reportl.frx TO PRINTER NOCONSOLE 


Report # 2 - Type of Training by Organization 


SELECT DISTINCT Reportl.unit, Reportl.tot, COUNT(Reportl.course_num)„ 
SUM(Reportl.cost), Traintyp.type, Units.unitname; 

FROM Reportl, Courses, Units, Traintyp; 

WHERE Courscs.course_num = Reportl.course num; 

AND Units.unit = Reportl.unit; 

AND Traintyp.tot = Reportl .tot, 

GROUP BY Reportl.unit, Reportl.tot; 

ORDER BY Reportl.unit, Reportl.tot; 


C^TCRETORTT^m ta FROM R«port2 WIDTH 5120 COLUMN NOOVERWRITE 
RETOCT repoS^ r„ TO PRINTER NOCONSOLE NOEJECT 
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07/15/93 1994 iiidiviclual Uevelopmcnl Tlaiis 


Orgaiilzatiun: ac -- comptroller 

ryiicdfTrninuig: lO - Executive and Management 


Klfl94 

K2176 

K514 

K545 

K6140 

K6166 

K6339 

K6427 

K6715 

K6777 

K753 

K0164 

K8176 

K9716 

K9941 

K9942 


<’««irse 'I Iclr 

cour EXEC run t'ot. issue 

SCIEHCE TECH AMP rtlPMC TOMCY 
NASA MArJACEMEIlT EtMTCATION PROG 
EXEC PEVEI.nrtlEtlT SRIINAR 
RES MGT E!''tIC PROGRAM 
MAtlAGEMElIT PEV SE?1 
IRiMAtl EI.ETIEtrr (THE) 
roMMPNTCATlOM h IPEI.UEMCE 

MEP VIPPATE;MGMT PRACTICES A ORGAM. CLIMATE 
PHPERSTAtlPItlG DUSIPESS POt.ICY S. OPS. 
MAtlAGEMElIT PEVELOPMEMT SEMI MAR 
APMltllSTRATIOM OF PUBLIC POLICY 
CROSSING PEPARITIEMT LIMES 
TASK MAMAGeiENT (NASA) 

SEMINAR ON MANAGERIAL COMPETENCIES 
EXECUTIVE PROJECT MANAGEMENT 


Subl»<a1sr«rl3|)c lo 


Executive and Management 


Number of Requests 
1 
1 

12 

1 

5 

1 

3 

3 
1 
2 

4 
3 
9 
3 
1 

_ 2 _ 

52 


Ci»s( 

$0.00 
$ 1 , 775.00 
$ 24 , 000.00 
$ 2 , 175.00 
$ 4 , 465.00 
$ 1 , 350.00 
$ 3 , 750.00 
$ 630.00 
$ 670.00 
$ 6 , 600.00 
$ 0.00 
$ 5 , 325.00 
$ 4 , 140.00 
$ 0.00 
$ 0.00 
$0 ■ 00 

$ 54 , 880.00 
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07/15/93 


Pao« 


1994 Itiilividual Dcvclopincnl Plans 

Ot gailizntioii: AC -- comptroller 


'l>|ic «f 'i rniliitif;: Number of Rrqnc.tti 


10 

-- 

Extent Ivp aruJ 

52 

20 


Suporv! f:ot y 

68 

3.1 

-- 

Enci 1 lfu| - OonornI 

3 

3M 

-- 

En<^ Itipot 1 net - Aprof^t^acp 

2 

30 

-- 

Fnqjnppr lnq ~ KlpcP.r leal/F.lpcttonic 

1 

3T 


Enqinnptlnq - Fyptpmrr, Inrltistrlal 

1 

4A 

-- 

ruhllc or nuRluppj^ Aclm!nl5;t. rat Ion 

46 

4D 


F1nancp, Accounting, Auditing 

09 

4C 

-- 

Pprsonnol 

9 

4n 

— 

EFO, Raro or Ftluilc Etndlpf! 

2 

4n: 

-- 

rrocnrpinf^tit 

97 

40 

-- 

Tolley, Ttogram, or Manngr^ment Analyaia 

7 

4M 

-- 

??pcurlty or tnvpjstJgat 1 on 

2 

4U 

-- 

Program and Projoet Managomont 

25 

4Z 

-- 

OtliPt Adm 1 n 1 f;trat 1 vo 

1 

5A 

-- 

.Systoins, Eafoty and Aroa Access Training 

4 

sn 

-- 

Acr-idoinlc Cotirso Work 

101 

6A 

-- 

Raslc Clor leal 

0 

6n 

-- 

Intor-porrjona 1 and of flee Managomont 

11 

6C 


Adminirtrat 1 VO System*? 

10 

6I> 

-- 

Othor Clerical 

2 

OA 

-- 

Orientation 

1 

RE 

-- 

Nots/Onallty Circles 

3 

er 


Productivity 

1 

m 

-- 

Professional/Personal Development 

13 

OH 

— 

Commiinication Skills 

25 

07. 

-- 

Other O«>noral 

1 

9A 

-- 

TelecommnnlcatIons, Not working 

2 

9D 

-- 

Prograrmnlng and Programming Languages 

26 

9D 

— 

office AutomatJon/AIM, Hus 1 ness 

4 

9R 

-- 

Software Managomont, Systoms Software 

2 

97 

-- 

other Computor llatrlwato aiul So ft warn 

7 


I 


$54.BnO.OO 
S37.7n4.00 
S325.00 
$2,550.00 
$ 0.00 
$390.00 
$13,136.44 
$33,332.00 
$960.00 
$ 0.00 
$24.B71.00 
$3,050.00 
$500.00 
$8,084.00 
$320.00 
$ 0.00 
$22,797.88 
$634.00 
$1,009.00 
$750.00 
$99.00 
$ 0.00 
$131.00 
$386 

$1,418.W' 
$2,438.00 
$595.00 
$1,210.00 
$1,255.00 
$2,500.00 
$1,127.00 
«:o no 


'I'UrALS I'OU AC -- Comptroller 


621 


216,532.32 


%■ 
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2,1.3 Report # 3 - Summary by Organization 

SELECT DISTINCT Report Limit, COUNT(ReportLcoursc_num),; 

SUM(Rcportl.cost), Unils.unitname; 

FROM Report 1, Courses, Units, Traintyp; 

WHERE Courses.coursenum = Reportl.coursenum; 

AND Units.unit = Report 1.unit; 

AND Traintyp.tot = Report 1. tot; 

GROUP BY Report 1.unit; 

ORDER BY Report I .unit; 

SSESlTCrtJ ftx FROM RCPOR3 WIDTH MO COLUMN NOOVERWRITE 
REPORT FORM report3.frx TO PRINTER NOCONSOLE NOEJECT 


2.1.4 Report # 4 - Summary by Type of Training 


SELECT DISTINCT Report 1.unit, COUNT(Reportl.course_nuin)„ 
SUM(Reportl.cost), Unils.unitnainc; 

FROM Reportl, Courses, Units, Traintyp, 

WHERE Courses.course_num = Report I.course_num; 

AND Units.unit = Reportl.unit; 

AND Traintyp.tot = Reportl.tot; 


GROUP BY Reportl.tot, 

ORDER BY Report!.tot; 

f- from RCPON4 WIDTH MO COLUMN NOOVERWRITE 
REPORT FORM report4.frx TO PRINTER NOCONSOLE NOEJECT 


2.2 On Training Costs and Priorities 

2.2.1 The Financial Impact of Training 

Training is expensive, however, lack of training could be even more expensivel Reliable 

sr„r.o" 

"ROOKS" and "OTHER COSTS" store information about each course s cost. Ui y, t-l „ 

bases.' 

. Direct Materials, instructor, tuition, cost for media, cost of employee travel and per diem, 
employee salary during training, opportunity cost of foregone production. 

. Indirect Administrative, ofTice space, computers, simulators, depreciation of facilitie^ 

. Iningihles. Costs associated with potential failure of personnel to p^orin a task or job, savings 
assoefated to finishing a job early due to good performance by everybody involved. 


j refer here lo iU -cplcj-H 

"direct” eost may understood as the cost lx,me by the trainees ocgan.z.Tt.o^ mdncct rs.nyc 

organizalion. 


rrtvt D n-r Alicnici T 1993. 
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Organizalitm 


1994 Individual Development Plans 
Summary by Organization 

Niimber of Requests 


page 


-- No mail code provided 
AC -- Comptroller 
CC -- Chief Counsel 
CD -- Center Director 

CM -- Payload Management and Operations 
(.p __ Payload Projects Management 

— STS Payload Operations 
CV Expendable Vehicles 

DE — Engineering Development 
— Facilities Engineering 
DL -- Electronic Engineering 
dm -- Mechanical Engineering 
EO -- Equal Opportunity Program 
EX Executive Management 

HM -- Human Resources and Management Systems 

MD — Biomedical Operations and Research 

MK - NASA Headquarters /space Shuttle Operations 

OP Procurement Office 

PA — Public Affsirs Office 

PM -- Personnel Office 

PT — Technology and Advanced Projects Office 
RM — Mission Assurance 
RO -- Quality Assurance 

RQ Reliability and Quality Assurance 

__ Safety and Reliability 
SI -- Center Support Operations 
gg Space Station Project Office 

TE Grounds Engineering 

TL -- shuttle Logistics Project Management 

TM -- Shuttle Management and Operations 

TP -- Shuttle Operations 

TV -- Vehicle Engineering 


Cost 


Totals for KSC 


13 

$4,764.00 

621 

$216,532.32 

7 

$2,389.00 

20 

$23,616.00 

40 

$18,154.00 

318 

$193,895.11 

2318 

$1,321,368.35 

170 

$121,802.00 

136 

$59,804.00 

900 

$484,378.92 

1033 

$572,131.68 

1225 

$459,930.46 

28 

$5,742.00 

55 

$19,647.00 

12 

$5,607.00 

294 

$148,244.18 

156 

$55,289.00 

710 

$301,214.91 

221 

$126,324.01 

733 

$103,086.00 

154 

$81,938.00 

490 

$294,545.' 

2464 

$548,692. iV^ 

50 

$30,072.66 

659 

$226,156.10 

1254 

$544,933.96 

262 

$159,045.00 

985 

$811,079.95 

363 

$180,313.00 

471 

$267,918.92 

842 

$338,121.95 

3066 

$1,388,009.52 

20070 

$9,114,747.17 
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Type of Training 


1994 Individual Development Plans 

Summary by Type of Training 

Number of Requests 


Cost 


10 

20 

3A 

3B 

3C 

3D 

3F 

3H 

3J 

3K 

3M 

3N 

3P 

3Q 

3R 

3S 

3T 

3U 

3W 

4A 

4B 

4C 

4D 

4E 

4G 

4J 

4K 

4N 

4P 

4Q 

4R 

4S 

4U 

4Y 

4Z 

5A 

5B 

5C 

6A 

6B 

6C 

6D 

7B 

7Z 

8A 

8B 

8C 

8E 


Engineering 

Engineering 

Engineering 


Executive and Management 

Supervisory 

Legal 

Medical and Biomedical 
Physical Sciences 

Earth Sciences and Natural Resources 
Human Factors 

Mathematics and Statistics 
Engineering - General 

- Aeronautics 

- Aerospace 

_ ^_ ^ - Chemical, Petroleum, or 

Engineering - Civel, Architectural, 
Engineering - Electrical/Electronic 
Engineering - Mechanics/Mechanical 
Engineering - R & QA 
Engineering - Systems, Industrial 
Engineering - Safety 

Other Legal, Medical and Scientific 
Public or Business Administration 
Finance, Accounting, Auditing 
Personnel 

EEO, Race or Ethnic Studies 
Procurement 

Policy, Program, or Management Analysis 

Quantitative Analysis or Operations 

Safety Systems and Operation 

Security or Investigation 

Logistics 

Supply 

General Safety and Health 

Foreign Language 

Program and Project Management 

Sales, Marketing and Customer Service 

Other Administrative 

Systems, Safety and Area Access Training 
Academic Course Work 
On-the-Job Training 
Basic Clerical 

Inter-personal and Office Management 
Administrative Systems 
Other Clerical 

Engineering and Science Support/Journeyman 
Other Trade, Craft, Apprentice and 
Orientation 
Career Planning 

Pre-Retirement Pile FIN. 

Nets/Quality Circles 


1211 

$1,892,420 

2691 

$1,655,615 

31 

$17,805 

33 

$4,620 

19 

$6,655 

112 

$65,637 

15 

$0 

5 

$178 

186 

$44,420 

33 

$4,322 

698 

$391,765 

146 

$56,735 

190 

$115,939 

473 

$358,319 

165 

$97,324 

246 

$128,603 

55 

$72,268 

100 

$58,419 

46 

$5,565 

1224 

$138,947 

249 

$75,729 

291 

$423,196 

272 

$16,439 

745 

$188,475 

53 

$14,938 

32 

$5,226 

34 

$12,407 

84 

$12,470 

59 

$22,669 

20 

$3,345 

288 

$59,557 

14 

$6,876 

1065 

$836,679 

22 

$5,545 

86 

$27,610 

1325 

$0 

2050 

$552,941 

24 

$0 

109 

$9,086 

354 

$35,828 

242 

$30,297 

107 

$9,989 

39 

$4,945 

7 

$7,000 

24 

$0 

243 

$6,836 

, August y. 5993, page 8 

$1,188 

472 

$29,117 
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1994 Individual Development Plans 


Summary by Type of Training 

Type of Training Number of Requests 


Cost 


8F 

8G 

8H 

8Z 

9A 

9B 

9C 

9D 

9E 

9F 

9Z 


Productivity 

Professional/Personal Development 
Communication Skills 
Other General 

Telecommunications, Networking 
Programming and Programming Languages 
Engineering Applications 
Office Automation/AIM, Business 
Software Management, Systems Software 
Artificial intelligence, Expert Systems 
Other Computer Hardware and Software 


67 

$73,907 

943 

$156,450 

1117 

$271,057 

85 

$68,633 

117 

$95,060 

764 

$710,244 

146 

$106,807 

586 

$66,532 

71 

$40,547 

6 

$1,875 

104 

$9,690 


TOTALS FOR KSC 


20070 $9,114,747 
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Flic: Rcporl#^ Suinmarj by Type of Training 


2.2.2 The Priority of Training 


PTDS- IDP.DBF daubase Me airrcally stores a -PRIORITY' neld for 
Uarodonalely, tire fcid redects the order in »l.icb Ihe «apl^ listed 

coorse listed is assigned priority 1 , the second one has ptionty 2. ete ). Obviotisly this does not 
potential impact of the training for the Center, nor for the person making the request. 


2.3 Recommendations for Further Work 
2.3.1 On the Financial Impact of Training 

Ensure that ^ 

ssro-^ 

Branch. The cost data base should store for wch course, t a 6 PJ ^ , computer will 

resources travel requirements, etc.), rather than the actual dollars spent. At repon time v 

2.3.2 On the Priority of Training 

tmdude in ihe da,a base a Held Uut, renccis the "t'"" 

has for lire Cenlet and for the person making 1 e leq ^ m- g„, u may be the best method to 

rrerattrvrs^ 

trainee. 

AS a possible indicato, of priori,y of eonien. and riming. "" 

Summer of 1992 in the Needs Assessment effort conducted by the author at KSC. 


Potential Impact 


Weight 


Helps organizational performance 
Helps individual’s performance in current position 
Enables person to perform more responsible position 
Useful but not essential 

There is no need for training in this subject mailer 


4 

3 

2 

1 

0 


Priority on Timing 


Weight 


Needed within next three months 
Needed witliin next six months 
Needed within next year 
Not an urgent need 


4 

3 

2 

1 
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3,0 THE EVALUATION OF TRAINING 


3.1 Existing Evaluation Procedures 

Evaluation of training courses is limited to KSC Form 13 (see next page). This form is processed 
manually and major problems are lira. 11 is not linked Co KSCs goals and “ „„ 

cosl^enefits considered, the form is not used to assess performance improvement, and the results are not 

used for formal reporting. 


3.2 Objectives 

The evaluation of training at KSC is envisioned as having the following properties: 

• Linked to KSC Mission, Goals and Objectives. 

• Has a perforniancc and valucs-based focus. j , 

• Measures results in terms of performance improvement, financial impact productivity 
and quality improvement, team building accomplishments, and less emp oyec urnover. 

• Resets priorities based on results. 

• Tracks training and performance data. 

Tlie link to KSCs mission, goals and objectives and the performance and values-based focus will 
be nscertT.;^ "hrongb nmr invelveuL, I, is the nse, (NOT the Human Resources D^ opment Branch) 
mho must define »hat is "required level of performance-. The 

Lining's impact on performance by asking the users: "What is the espected result of tratntng? and 
"What may happen if no training is given?" 


3.3 Evaluation: A Literature Search 

A literature search was performed to ensure that the proposed evaluation tvd^l 

standard practice. The following sections discuss the souths, most common 

methods to conduct the evaluation and an indicator of performance elfectiveness, the Content Validity 
Ratio" A detailed discussion of the search findings is provided in the appendix, along with the ^ 
mfelces u1^. Copies of alt references were submitted to NASA in a separate binder and are considered 

part of this final report. 


3.3.1 


Sources 


The literature search yielded 782 hits when the keep words "TRAINING", "EFFECTIVENESS , 
"AS<;eSSMENT" and "EVALUATION" were used. From those 782 titles, a total of 78 abs‘racls were 
se^tL aiX^nJ m of those abstracts resulted in 22 promising articles of which H-e 

ava^abk at UCFs library (or though inter-library loans). Most articles are rather theoretical in nature but 
a few described specific applications in government agencies such as the 

Protection and Quarantine and Forest Service divisions), NASA Headquarters, the Department of 
Navy’s Navy Finance Center, and the Office of Personnel Management. 
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PARTICIPANT'S EVALUATION 



PARTICIPANT'S NAME 



SUPERVISOR'S NAME 


PLEASE INDICATE YOUR EVALUATION OF THE SEMINAR IN TERMS OF THE FOLLOWING: 


EXCELLENT 


1. Ov€Col1 gyoluQ tion o( fminoi. - - 

2. S*minof content - omeont o\ Jetoil onJ enfnt of covfoge. 


3. Level ol pre»enlollon. __ _ 


4. E Necti venex ol hon4~oul motefioU. 


5. E If ecli vene** ol oodio/vimol oiJi. 


6. Inilfuclof(A) ttnovyledge of tubjecl. ___ 


7. ln«truclOf(i) efiectivene i*.___ 


8. Applicotion to your fob. 



COMMENTS: 
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3.3.2 Approaches 

Trai„ing«a,»a,ion„su,UyUkc.c.«of^^^^ 

conductcdatlhecndoflhctrainingexcrcis /.UMrnine where Ihcevalualionfocusesoawhelhcf 

envirenmeut. a";l f “”^1 .he suhieC h.„gh,; (3) 

—«ns.io„ foiiowfug 

benefits accnied as a result of the training. 

3.3.3 Methods 

to achieve as a result of the training and --views inauiries and the PAPA method are tliat they 

attained. The main problems with ‘ and’money) to conduct. For an on-going 

.rmy lack objectivity and require vast resource* the best 

'.2i.^Srs sSrdiXU which printing, disIribnUon. eoUecion, analysis and 

reporting can be mechanized. 

3 3 4 Content Validity Ratio (CVR) 
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3.4 The Evaluation Procedure 

The proposed evaluation procedure consists of a set of Uvo evaluations and three reports. Copies 
of the evaluations (as proposed and as implemented) arc included in the next pages. 


3.4.1 The Scanning Process 

A„OpScanModel 5 r,o.,iNalionalCompuler Syaems(NCS)««J”’’' 
l„3 .o nKctoaiao „ 

design, a Windowa based i„ forms sepplied by 

3.4.2 Evaluations 

The proposed End-of-Course Evaluation is a "reaction" type evaluation. It h« four sertions. The 
firs.one, -Exlernals,- deals teU 'S^menl.- 

■'“'■'3S me IrS f^J^rSe pemeSalue for’ .be organizalion. An <»en,l. -Happiness Index- 

A Follow Up Evaluation was proposed, to be delivered about six months after the training. Both 

supervisors’ delayed assessment of the real impact of each training course. 


3,4.3 Reports 

under his supervision. 

An End-of-Quarter Report is planned which will summarize tlie number of purses taught, the 
number of students, cost, and indicators of overall happiness and 3 ,„3 ,, 
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KENNEDY SPACE CENTER 
TRAINING BRANCH 

INITIAL EVALUATION OP TRAINING 


SSNs 

• 

HAIL CODE! 



COURSE! 
INSTRUCTOR! 
COURSE DATB(S)i 
SITE! 


INSTRUCTIOMS: Indicate the degree to which you agree with the following =««”ie"'s using 
steSTI!OHGl.Y agree a=AGREE n.DISAGREE »D=STROKGLY DISAGREE ia=NOT APPLIC 


EXTERNALS 

1. The environment was suitable for learning. 

2. There were no distractions during the session. 

3 Handouts used at the session were effective. 

4. Audio/visual aids used at the session were effective. 

INSTRUCTION 

5. The instructor delivered the lessons clearly and effectively. 

6. The instructor was knowledgeable of the materia taug 

7 The instructor kept my interest throughout the session. 

8. The instructor showed concern for the student’s understanding 

of the material. , p.- .,1,. 

9 . The hands-on component was delivered effectiv y. 

10. Enough time was alloted to hands-on practice. 


SD NA 




FOCUS 

11. The material in this class is consistent with the actual and 

current reguirements of my job. 

12 . The training focused on the specific tasks, knowledge an 

skills needed for acceptable job performance. 

13. The knowledge and skills were taught to the appropriate le 

of proficiency. 

14 Training was included that is not needed on the 10 b. 

15 . Required skills and knowledge were not adequately covered. 

16. The level of knowledge attained in training meets the level 
needed for acceptable job performance. 


00000 

00000 



00000 


assessment 

17. Overall, I am pleased with this course. 

18. This training will help me on my job. 

19. This training course is right on target with KSC s 

20. I raconmend that othara from my oroanlzatlon attand thla 


needs. 

courso. 
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KENNEDY SPACE CENTER 
HUMAN RESOURCES DEVELOPMENT BRANCH 

INITIAL EVALUATION OP TRAINING 


o 


NAME: 


COURSEI 
INSTRUCTOR i 
COURSE DATE{S)i 
SITEt 


INSTRUCTIONS: Indicate the degree to which you agree with the following statements using 
SA=STRONGLY AGREE A=AGREE D=DISAGREE SD=STRONGLY DISAGREE NA=NOT APPLICABLE 


PLEASE COMPLETELY FILL IN THE APPROPRIATE BUBBLES WITH A #2 PE!NCIL. 


EXTERNALS 

1. The environment was suitable for learning. 

2. Handouts used at the session were effective. 

3. Audio/visual aids used at the session were effective. 



INSTRUCTION 

4. The instructor delivered the lessons clearly and effectively. 

5. The instructor was knowledgeable of the material taught. 

6. The hands-on component was delivered effectively. 



FOCUS 

7. The material in this class is consistent with the actual and 
current requirements of my job. 

8. The level of knowledge attained in training will help improve 
my job performance. 


ooooo 

ooooo 


ASSESSMENT 

9. Overall, I am pleased with this course. 

10. This training course is consistent with KSC*s needs. 

11. I would recommend that others from my organization attend 
this course. 



OOOOO 


Suggestions for improvement or other comments: 
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KENNEDY SPACE CENTER 

training branch 

FOLLOM-tJP EVALUATION OF TRAIHINO 



SSNi 

- 

HAIL CODEt 



COURSEt 
INSTRUCTORt 
COURSE DATE(S)t 
SITE: 


,NSTEUCTI0»S, indicate the degree to which you agree with the following statements using 
sa^SThONGLY AGREE A^AGREE n=DISAGREE sn=STRONGLY DISAGREE HA=NOT APPLICABLE 


On the basis of this employee's performance after the referred 
?rain!nrcourse, the following have improved: 


1. employee's supervisory and management skills 

2. personal productivity. 

3. organization's productivity. 

1. employee's satisfaction. 

wwj. customer's satisfaction. 


6. job performance-quality. 

7. job performance-quantity. 

8. organization's morale. 

9. safety. 

10. stress reduction. 

11. teambuilding. 

12. cost awareness and control. 

13. environmental awareness. tfco-c; needs 

14 This training course is right on target with KSC s needs, 

is! I r.c«-»4 that othar. tram w otgaalaatlou attaad thl. ooutaa. 


SA A D SD NA 



THIS FORM NAS FILLED OUT BY: 


employee 

SUPERVISOR 


TODAY'S DATE! 
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TO: 


(SUPERVISOR’S NAME} 


MAILCODE: 


o - 


KENNEDY SPACE CENTER 
HUMAN RESOURCES DEVELOPMENT BRANCH 
FOLLOW-UP EVALUATION OF TRAINIHO 


PARTICIPANT'S NAME: 


COURSE t 
VENDORt 

COURSE DATE(S)i 


degree to Shich you agree with the following statements using: 


SA=STRONGLY AGREE 


A=AGREE D=DISAGREE SD=STRONGLY DISAGREE NA-NOT APPLICABLE 



1. This employee's job performance has improved as a direct 
result of this training. 

2. I would recommend that others from my organization attend 
this course. 

TODAY'S DATE: __ 

Suggestions for improvements or other comments concerning 
this training: 


ooooo - 
ooooo - 


PLEASE RETURN THIS SURVEY TO HM-PER-1 IN A U.S. GOVERNMENT 
MESSENGER ENVELOPE. 
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3.5 Expectations 

The proposed evalualioi. procedure will (a) and 

Llualion of Ihc results obtained (aggregate values). 


3.6 


Recommendations for Further Work 
3.6.1 On Evaluation: 


The firsl fccomnicndalion is lo “I*™'® 

report recipients. 


3.6.2 


On the Training Management Information System: 


^ Vyll —- cr - 

«t PTDS rNTDS?) informalion 
The second recommendation is to link the hours, units of resources etc.) (b) 

data. 


3.6.3 On the Needs Assessment: 


V/li -- 

The whole Training System should be (Needs Analysis), how 
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A O APPPNDIX: Literature Review 


T,„ r„n.»in6 pages psey,^ e “<ST 
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training evaluation - An Overview Of Existing Practices 

Four categories of training evaluation are reaction, 
learning, behavior and results [Stevens & Hellweg, 1990]. 

Reaction is the trainee's liking of the training 
course. This response should be determined as soon as 
possible after the session is completed [Dewine, 1987]. 
Questions dealing with the atmosphere, physical environment, 
instructors and training media should be included on the 
first questionnaire. An example of a reaction based 
question is: 

6. Rate the use oC handouts in the training. _Excellent .Good .Fair .Poor 

Learning is the evaluation of whether the trainee 
remembers (not necessarily uses) facts, principles and 
behaviors taught in class. This is the least relevant 
level of evaluation for an organization because increased 
knowledge does not necessarily mean the knowledge is used on 
the job [Stevens & Hellweg, 1990]. An example of a learning 

based question is: 

2. Pentium class microprocessors must be super-cooled prior to use. .True .False 

The behavior level is the most complex level of 
evaluation. This deals with the actual job improvement of 
the trainee. The behavior evaluation should come from 
trainee, its supervisor, subordinates, and peers. It should 
be performed at least three months after the training 
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[Stevens & Hellweg, 1990]. An example of a behavior based 
question is: 

6. I am capable of speaking to large groups: ^Excellently _well ^Acceptably ^Poorly 

Results are the actual effects that training has had on 
an organization. Items such as turnover, absences, sales, 
customer satisfaction, quantity and quality of performance, 
morale, cost reductions, grievance reductions, accident 
rates, suggestions, employee satisfaction and time use 
should be evaluated [Stevens & Hellweg, 1990]. 

8. Morale at the worksite has (since my training). _Increa6ed significantly 

_Remained about the same 
_Decreased significantly 

The outcome of one of the four areas may affect other 
areas [Dewine, 1987]. The amount of time between training 
and a post training evaluation is not a standard. Ranges of 
suggested times are from two weeks to twenty-three months 
[Stevens & Hellweg, 1990]. An average time is from four to 
six months. 

An alternate method of evaluation of training is 
content evaluation. Content validity is the evaluation of 
how the training content is related to the job. The 
training content must be identified. Items that are 
included are knowledge, skills, abilities, and other 
personal characteristics(KSAOs) that are needed for job 
pei^formance. The content is then evaluated through a Content 
Validity Ratio (CVR) approach. People who have a thorough 
knowledge of the job rate each KSAO on its importance to the 
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job. The following formula is used to determine validity 
[Ford & Wroten, 1984]. 

CVR PPonl<> fil-atlno KSAO Is Imnortant. - Peop l e Stat i ng KSAQ I S UUlmPOrta il L 

Total Numbar o£ People 

Positive CVRs mean that greater than half of the people 
stated that the KSAO was important and negative CVRs mean 
that less than half of the people stated that the KSAO was 
important. A method for matching the CVR data and training 
needs would include the following steps. First divide the 
KSAOs into categories. Have your job experts rate the 
important KSAOs on a scale. This scale would determine how 
training intensity and resources should be'spent on 
different areas of subject content [Ford & Wroten, 1984]. 
These methods would work well. However, they would require 
customized forms for each course and extra manpower because 
the procedure could not be automated. 

Kruger & Smith (1987) recommended two different 
surveys: a skills/behavior survey and a health/stress 
survey. Both surveys use a system of examining statements 
and choosing to what degree you agree with the statements. 
The following is an example: 

3. I am able to communicate with my manager. AGREE 987654321 DISAGREE 

The skills/behavior survey deals with what was 
learned, how the trainee’s performance has changed and how 
it has affected the organization. A survey is filled out by 
the trainee, its supervisor and some others. A sample 

nic: WRTl)PTOT.IK)r, 22 Julj, 1993, page 

457 



question on a skills/behavior survey might be: 

6. I am capable of speaking to large groups: _Excellently _Well _Acceptably _Poorly 

The health/stress survey identifies health and stress 
risks to the employee. It is only be filled out by the 
trainee. A sample question on a health/stress survey might 

be: 

6 My knowledge of safety that is required for the job Is: _Proflcient 
^ _ Adequate 

_Not Adequate 
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Existing 


Training Evaluation Programs 

t-raining evaluation programs 
The following are some training ev 

1 =>i- fiifferent governmental and non- 

currently in place at difterenc g 

governmental agencies. 

pepartnent of Agriculture-Plant Protection ana 
ouarantlne: The Department of Agriculture. s Plant Protectron 
ana Quarantine branch currently uses an evaluation program 

in which the evaluation is done by front-line supervrsors. 

^ an t-Vifa skills needed to 
•oorci are first trained m tne sku 
The supervisors are 

evaluate the trainees. The supervisors are then assigned to 
evaluate recently trarned officers. The evaluators may use 
questionnaires, interviews, performance observations an 
miscellaneous inguiries to gather data. These are all use 
Che training. The trainees supply data concerning 
Cheir performance through either guestionnaires and 
,„cerviews. The evaluators gather their own data through 
pn-the-iob performance evaluations and candid inguiries^ T 
aata is collected by a central organisation and compile 

if the skills taught actually assisted the 
determine it the sxin 

■th their jobs. Advantages of this program mclu 
officers with tneir juuo 

1 ant data for evaluation and actively 
providing relevant data 

involving supervisors in the evaluation. Disadvantages 
ipclude long length of time needed for process, regurring 
supervisors to do wor. solely on evaluation causes need t 
hire temporary replacements and expensive travel costs o 
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supervisors. A final disadvantage is that the evaluation is 
based on the specific job or supervisor's objectives and not 
those of the entire organization (Salinger & Roberts, 1984] 
Department of Agriculture-Forest Service: The Forest 
Service currently uses a questionnaire-based evaluation 
system. Standard questionnaires are completed by the 
trainee, its supervisor and subordinates before the training 
session and four weeks after the session. Copies of the 
questionnaire are also completed by untrained personnel and 
their supervisors and subordinates in order to have a 
control population. Questions in the questionnaire are 
behavior based such as the following 


To what extent do you think training: 

will improve your ability to be a better supervisor 

Will increase productivity? 

Is a good use of your time? ^ 

Improve your communication skills on the job? 


GREATLY NOT AT ALL 

987654321 
987654321 
987654321 
987654321 


Results are statistically tabulated. The results are then 


compiled and reported to the specific organizations. The 
questionnaires are behavior based and completely 
confidential. The tabulation stage tests the statistical 


significance of questions, trainee, supervisor and 
subordinate perceptions and comparison of trained results 
versus the control's results. Advantages of this program 


include accurate results due to use of a control group and 
small amounts of time and resources needed. The only 
significant disadvantage is the possibility of low return 
rate of questionnaires [Salinger & Roberts, 1984]. 
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Administration: The 


National Aeronautics & Space 
evaluation program described below was for a career seminar 
sponsored by NASA Headquarters. The evaluation was directly 
built into the training program. The evaluation consisted of 
both pre and post training questionnaires, group interviews 
and individual interviews. The questionnaires contained 
general, open-ended questions dealing with current and 
future dob goals, A sample question on a survey might be: 

7 „„.t h... a=n. to .chi... ,o»r ,..1= In th, p..t 30 O.y.t 

in the group interviews, the trainees spoke about their own 

dob plans and how they were using what they learned. 

Trainees could hold optional individual interviews with the 

instructor concerning their own progress. There were also 

informal evaluations during the course. The instructor was 

the only evaluator involved and there was no mador 

computation of data in the evaluation. The instructor had to 

be available after the training session for many months in 

order to do the personal interviews. Advantages of the 

evaluation program included adaptability. Disadvantages 

include the fact that it is not very structured, forms need 

to be customized for different courses, constant need tor 

the instructor, no organizational input and most of work has 

to be in groups which may be difficult to assemble [Salinger 
& Roberts, 1984]. 

Department of the Navy- Navy Finance Center: The Navy 
Finance Center evaluated one of its Interaction Management 
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Courses through a system of questionnaires done in a group 
setting. The questionnaires were completely behavior based 
with questions such as: 

When my eupervlsor speaks to me regarding disciplinary action , he/she ... 

Always Never 

Uses my Ideas when I offer them. 654321 

Acknowledges my feelings about the situation. 654321 

The questionnaires were completed by employees, their 
managers and subordinates that did and did not take the 
course. The evaluation was done before the training and two 
years after. The results were then computer tabulated to 
create statistical averages of how the training works 
through the perceptions of the employees, their managers and 
subordinates. Reports were then created from that data. 
Advantages of this evaluation method includes involvement of 
all members of an organization, use of a control (the 
untrained sample) to ensure accuracy of results and high 
return rate because of the group approach to completing the 
questionnaire. Disadvantages include a management commitment 
and travel expenses if the respondents are not in a common 
geographical area [Salinger & Roberts, 1984]. 

Office of Personnel Management: The 0PM devised a 
generic training evaluation program called the Participant 
Action Plan Approach (PAPA). The evaluation system asked 
each participant to identify certain behaviors or actions 
related to the course that he/she would like to change. Here 
the evaluator would either interview the trainee (by phone 
or in person) or use questionnaires (six months after the 
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training) to determine whether the actions were carried 
out. The evaluation also checired for results of tried action 
plans and why untried action plans were not attempted. The 
questionnaire also as)ced how the course could be changed to 
better teach the trainee. Problems and obstacles that 
deterred the trainee from carrying out the plan were also 
identified. The result were computer tabulated. Results were 
used to create reports and eventually modify the course. The 
program provides data primarily concerning behavioral 
changes brought about by the course. Advantages of this 
program include ability to be used for many courses and it 
can accurately describe what actions came from the training. 
Disadvantages include the need of self-reports and skilled 
interviewers, large amount of time needed and it does not 
involve supervisors and subordinates in the evaluation 
process [Salinger & Roberts, 1984]. 
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